Autoencoders

An autoencoder is a type of artificial neural network used to learn efficient codings of unlabeled data (unsupervised learning). The encoding is validated and refined by attempting to regenerate the input from the encoding. The autoencoder learns a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore insignificant data (“noise”)

Resources

• https://en.wikipedia.org/wiki/Autoencoder
• Dimensionality Reduction
• The classical approach for unsupervised learning using neural networks. The basic version consists of a Multilayer Perceptron (MLP) where the input and output layer have the same size and a smaller hidden layer is trained to recover the input. Once trained, the output from the hidden layer corresponds to data representation that can be useful for clustering, dimensionality reduction, improving supervised classification and even for data compression.
• https://blog.keras.io/building-autoencoders-in-keras.html
• https://blog.insightdatascience.com/isee-removing-eyeglasses-from-faces-using-deep-learning-d4e7d935376f
• https://github.com/nanopony/keras-convautoencoder
• Meta’s AI Takes an Unsupervised Step Forward

VAEs

• Variational autoencoders are generative models. Traditional autoencoders that just do reconstruction don’t have an obvious generative interpretation. There are some cases in between, like denoising autoencoders, where it is possible to construct a Markov chain that uses the autoencoder to sample from the data distribution, but the autoencoder doesn’t give direct explicit access to an estimate of the density or the ability to sample directly.
• VAE is a type of autoencoder with added constraints on the encoded representations being learned. More precisely, it is an autoencoder that learns a latent variable model for its input data. So instead of letting your neural network learn an arbitrary function, you are learning the parameters of a probability distribution modeling your data. If you sample points from this distribution, you can generate new input data samples: a VAE is a "generative model".
• Introduction to deep generative modeling: Variational Auto-Encoders
• Introduction to deep generative modeling: Priors in VAEs
• Introduction to deep generative modeling: Hierarchical VAEs
• Variational Autoencoders Explained
• Intuitively understanding VAEs
• An Intuitive Comparison of Autoencoders with Variational Autoencoders
• http://blog.fastforwardlabs.com/post/148842796218/introducing-variational-autoencoders-in-prose-and
• Variational Autoencoders (youtube)
• From Autoencoder to Beta-VAE

Code

• #CODE benchmark_VAE
  • This library implements some of the most common (Variational) Autoencoder models
  • https://pythae.readthedocs.io/en/latest/
  • #PAPER Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case (Chadebec 2022)

References

• #PAPER Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion (Vincent 2010)
• #PAPER Adversarial Autoencoders with Constant-Curvature Latent Manifolds (Grattarola 2018)
• #PAPER Image-To-Image Translation Using a Cross-Domain Auto-Encoder and Decoder (Yoo 2019)
  • Early image-to-image translation methods used convolutional neural networks (CNN), which learn to minimize the loss of a pixel value between the source domain image and the target domain image but had the limitation of failing to produce more photorealistic images
  • Unlike other approaches… our method is not limited to a specific task, nor do we rely on predefined relationships between the source and target domains. Our method can be applied to make a general-domain solution for many image-to-image translation tasks.
• #PAPER Accelerated Design and Deployment of Low-Carbon Concrete for Data Centers (Ge 2022)
  • use conditional variational autoencoders (CVAEs), a type of semi-supervised generative artificial intelligence (AI) model, to discover concrete formulas with desired properties
  • https://www.theregister.com/2022/04/29/meta_ai_green_concrete/

Variational Autoencoders (VAE)

• #PAPER Auto-Encoding Variational Bayes (Kingma 2014)
• #PAPER An Introduction to Variational Autoencoders (Kingma 2019)
• #PAPER NVAE: A Deep Hierarchical Variational Autoencoder (Vahdat 2020)
  • Paper explained
• #PAPER Deep Attentive Variational Inference (Apostolopoulou 2022)

Masked Autoencoders (MAE)

• #PAPER Masked Autoencoders Are Scalable Vision Learners (He 2021)
  • #CODE https://github.com/facebookresearch/mae
  • #CODE https://github.com/ariG23498/mae-scalable-vision-learners
  • https://keras.io/examples/vision/masked_image_modeling/
  • Masked autoencoder (MAE) is a simple autoencoding approach that reconstructs the original signal given its partial observation
  • Paper explained
  • See AI/Deep learning/Transformers
• #PAPER Masked Autoencoders As Spatiotemporal Learners (Feichtenhofer 2022)
• #PAPER Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality (Li 2022)
  • #CODE https://github.com/implus/um-mae
• #PAPER ConvMAE: Masked Convolution Meets Masked Autoencoders (Gao 2022)
  • #CODE https://github.com/Alpha-VL/ConvMAE
• #PAPER Masked Autoencoders that Listen (Huang 2022)
  • #CODE https://github.com/facebookresearch/AudioMAE